Towards an Evaluation Methodology of Diagnostic Accuracy for Ill-defined Domains

Researchers agree that error diagnosis is one of the most important components of an Intelligent Tutoring System (ITS). Since in ambitious domains a perfect error diagnosis can not be guaranteed, the diagnostic accuracy of an ITS within an ill-defined domain should attract more attention. In this paper we introduce our constraint-based error diagnosis approach for logic programming and demonstrate an evaluation methodology which measures diagnostic accuracy and is comprised of two parts: evaluation of intention analysis and evaluation of diagnostic reliability.